Sparse Code Formation with Linear Inhibition

Sparse code formation in the primary visual cortex (V1) has been inspiration for many state-ofthe-art visual recognition systems. To stimulate this behavior, networks are trained networks under mathematical constraint of sparsity or selectivity. Meanwhile, there is another line of research which emphasizes the role of lateral connections of neural networks in sparse code formation. Lateral connections are synapses among neurons on the same layer, which is an essential part of human neural networks. There are two types of interconnections. Excitatory connections propagate firing signal across neural layer, thus preserve topographical order of neural stimuli. On the other hand, inhibitory connections decorrelate activations among neurons, which accounts for sparse code formation.